The present invention relates generally to a system for electronic music performance. More particular still, the invention relates to a system for permitting participants to collaborate in the performance of music, i.e. to jam, where any performer may be remote from any others.
Not Applicable
Not Applicable
The following two appendices are comprised of the respectively listed files, all of which are included on the compact disc filed with this application, and incorporated herein by reference:
APPENDIX Axe2x80x94exemplary application implemented in Java Studio, by Microsoft Corporation, using calls to SERIALIO, a serial interface class library by Solutions Consulting, Inc., of Santa Barbara, Calif. and the ActiveX Seer Music Player by Seer Systems of Los Altos, Calif..
APPENDIX Bxe2x80x94exemplary application implemented in Visual Basic, by Microsoft Corporation, using calls to their DirectX version 8.1 API, specifically the DirectPlay and DirectMusic components.
In earlier times, a musical education was considered essential. Today, researchers such as Raucher, Shaw and Ky at the University of California, Irvine, study the Mozart Effect (Nature, vol. 365, pg. 611), and are finding that musical training enhances brain power, especially in spatial reasoning. Congresswoman Louise Slaughter further observed, supporting a rededication to music and art training, that xe2x80x9cThose who create do not destroy.xe2x80x9d Nonetheless, music has been deleted from most elementary and high school curricula and dropped from many extracurricular programs. Private music lessons are expensive and many individuals lack the interest that book learning alone would require.
Personal computers and video game machines are found in most households. A growing fraction of these machines are able to interconnect using modems or the Internet.
Software has long been available to allow a computer to become a musical instrument and to provide music theory instruction. Practica Musica (1987), by Ars Nova Software, Kirkland, Wash. is an example of such. This program was sometimes bundled with a plastic keyboard overlay that would temporarily convert a computer keyboard into a miniature white-and-black-keys piano keyboard.
A drawback of such music programs is that they only admit one person at a time. It is desirable to allow students to receive music education using their computers, but to allow multiple students to play together. Further, by allowing the multiple students to be at remote locations (e.g. each in their own home), geography and transportation cost and time cease to be a barrier. Such a capability would allow an online community of music students to interact and collaborate. One should anticipate the formation of xe2x80x9cVirtual Garage Bandsxe2x80x9d and the creation of songs by composers and lyricists who have never met in person.
Historically, computer games only operated for a single player at a time, or for multiple players only at a single location, sharing a single computer. However, there is now a burgeoning market for multi-player games. Individuals with computers or video game machines at separate locations can connect via phone lines or the Internet and cooperate or compete in a computer game. One example of such a game is MechWarrior (1995) by Activision, which allows players"" computers to connect via phone lines. Another example is EverQuest (2001) by Sony Digital Entertainment, where many hundreds of players, each with a computer, connect via the Internet, to a game server owned and operated by the publisher, to play in the same game.
A key difficulty in designing multi-player computer games is the communication latency that occurs between the players"" computers. This results in a computerized version of the children""s argument, xe2x80x9cI tagged you first.xe2x80x9d xe2x80x9cNo you didn""t, I tagged you first.xe2x80x9d Two separated computers each accept their own (local) user""s input. The computers then communicate those inputs to each other, and finally use both users"" inputs to perform a game calculation. However, unless latency (delay) in the communication channel is managed in some way, each computer gives its local user a reaction-time advantage because the other (remote) user is always penalized by the communication channel delay. Eventually, this can result in a disagreement between the two computersxe2x80x94xe2x80x9cI tagged you first.xe2x80x9d
A number of methodologies, each having various virtues and drawbacks, have been developed to solve the communication latency issue for multi-player gaming.
Matheson, in U.S. Pat. No. 4,570,930 teaches a method for synchronizing two computer games. Applicable only to games having discretely calculated xe2x80x9cgenerationsxe2x80x9d, Matheson provides that each generation is numbered, and each generation calculation uses the users"" inputs gathered during the prior generation. Matheson""s generations may, at best, each be {fraction (1/30)} of a second long, i.e. at the game""s video update rate. However, generations can become arbitrarily long if one or another user""s input is not communicated in a timely manner, or needs to be retransmitted. Unfortunately, such a behavior is not conducive to musical performance.
Hochstein, et al., in U.S. Pat. No. 5,292,125, unlike Matheson, shows a technique for continuous play, not requiring Matheson""s xe2x80x9cgenerationsxe2x80x9d. Hochstein measures the roundtrip communications transport time between two game stations. Subsequently, each user""s input to their local game station is delayed by half the round trip time, but is transmitted to their opponent""s station immediately. This meets a xe2x80x9cfair gamexe2x80x9d criteria that Hochstein proposes, by which neither player enjoys a speed advantage over the other.
While the xe2x80x9cgeneration-lessxe2x80x9d technique is more conducive to musical performance, Hochstein does not address three issues. First, Hochstein does not account for unreliability of the communication channel. Second, simultaneity of players"" input is necessary but not sufficient to ensure that game stations remain synchronized. There is the asynchrony of the game station""s main loop which will cause divergence in the game state. Matheson understood this. Third, the xe2x80x9cfair gamexe2x80x9d criteria calls for system performance to be degraded to the lowest common denominator. In U.S. Pat. No. 5,350,176, Hochstein, et al. provides synchronization codes which addresses only the first of these. In doing so, he has nearly reverted to Matheson""s generations. Bakoglu, et al., in U.S. Pat. No. 5,685,775 provide an alternative synchronization, but at the expense of incurring the entire roundtrip communication transport delay, rather than only a portion.
O""Callaghan is the first to provide the xe2x80x9cfair gamexe2x80x9d criteria for more than two remote stations. Under his U.S. Pat. No. 5,820,463, a collection of two or more stations algorithmically selects a master. All inputs from all stations are sent to the master station, and all are subsequently sent out to each of the other stations for processing. Key drawbacks here, are just as above: performance is degraded to the least common denominator, and a latency of the full roundtrip communication transport delay is incurred.
In U.S. Pat. No. 6,067,566, Moline teaches a method whereby a live musical performance, preferably encoded as well known Musical Instrument Digital Interface (MIDI) commands, can be sent over a network to many stations. The live performance can be selectively recorded or mixed with other pre-recorded tracks. The mechanism is a timestamp that is attached to each musical event (e.g. a MIDI Note-On command). By sequencing the timestamps from separate tracks, the tracks can be mixed. By delaying the mixing for at least the maximum expected delay of the communication channel, the (almost) live musical performance can be added to the pre-recorded tracks at a remote location. Further, a station receiving this performance can play along with the (almost) live performance. Moline is limited, however, in that the xe2x80x9cplay alongxe2x80x9d performance is not bi-directional. That is, a true jam session is not taking place. Moline suggests that a repetitive musical pattern could be established and enforced, and that jamming could take place by having each participant hear and play along with the others"" performance from one or more prior cycles of the pattern. That play along performance is what would subsequently be heard by the others, during the next (or later) cycle. Such a constraint severely limits the range of artistic expression.
Rocket Network, Inc, of San Francisco, Calif., (www.rocketnetwork.com) allows a similar collaboration model, but without a true real time component. Through their Res Rocket 1.4 MIDI Collaboration software, a player can retrieve a MIDI sequence from a central server, subsequently play along with it and add or modify selected parts, and upload the additions or changes to the server for other collaboration partners to download in turn.
Tonos Entertainment, Inc, of Culver City, Calif., (www.tonos.com) provides a similar capability, but is based on MP3 files, rather than MIDI.
Neumann, et al. in U.S. Pat. No. 6,175,872 does not have such a limitation. By requiring synchronization to a single clock, time stamping of MIDI packets, the streams of MIDI data generated at remote locations can be sent to the local station, sequenced into the correct musical order, and played aloud. The participant playing on the local machine similarly transmits his musical performance, with timestamp, to the other stations. By further requiring that communication transport delay shall be less than twenty milliseconds, Neumann provides real time, remote collaboration and jamming. However, the twenty-millisecond constraint is not met for dial-up users to the Internet, nor even with a direct connection between callers on most local telephone exchanges. Further, the physical limits of the finite speed of light in fiber or cable accumulates at roughly 1 millisecond per 100 miles. Such a requirement, Neumann admits, limits collaborative jamming to a campus-sized WAN. Still, Neumann""s contribution allows the merger of multiple remote MIDI streams in another play along environment.
Thus, there is a need for a system that allows multiple musical players, especially music students, to collaborate and jam in real time and over useful distances, such as across neighborhoods, cities, states, continents, and even across the globe.
Because of the delays inherent in communication over significant distances, a technique is needed which does not compound that delay.
Further, there needs to be a way of limiting the adverse effects of excessive delay, and to allow each station to achieve an acceptable level of responsiveness.
The present invention satisfies these and other needs and provides further related advantages.
The present invention relates to a system and method for playing music with one or more other musicians, that is, jamming, where some of the other people are at remote locations.
Each musician has a station, typically including a keyboard, computer, synthesizer, and a communication channel. The communication channel might be a modem connected to a telephone line, a DSL connection, or other local, wide, or Internet network connection.
When musicians desire a jam session, their respective station computers communicate with each other, or perhaps with a designated host computer, and determine the communication delays to and from each other station in the jam.
Subsequently, each musician""s performance is immediately transmitted to every other musician""s station. However, the performance is delayed before being played locally.
Upon receipt, remote performances are also delayed, with the exception of the performance coming from the station having the greatest associated network delay, which can be played immediately.
The local performance is played locally after undergoing a delay equal to that of the greatest associated network delay.
By this method, each musician""s local performance is kept in time with every other musician""s performance. The added delay between the musician""s performance and the time it is played, becomes an artifact of the instrument. As such, the musician is able to compensate for it and xe2x80x9cplay aheadxe2x80x9d or xe2x80x9con top ofxe2x80x9d the jam beat.
Sometimes, some of the stations may have a low (good) communication delay between them, while others may have a high (bad) delay. In such a case, each musician can choose to have his station disregard high delay stations during live jamming, and to allow performance with only low delays.
In addition, a xe2x80x9cgroovexe2x80x9d can be distributed to the stations. The groove is a track that provides a framework for the jam session. In its simplest form, it might be a metronome. But it could also be a MIDI sequence, a WAV file (instrumental or lyrical), or an MP3. Regardless of the communication delays, the groove plays in synchrony on all machines, as the command to start the groove is delayed appropriately by each station receiving the play command, and the station issuing the play command.
It is the object of this invention to make it possible for a plurality of musicians to perform and collaborate in real time, even at remote locations.
In addition to the above, it is an object of this invention to limit delay to the minimum necessary.
It is an object of this invention to incorporate the artifacts of communication delay into the local performance in a manner which can be intuitively compensated for by the local musician.
It is a further object to permit each musician to further limit delay artifacts, to taste.
It is a further object of this invention to provide a groove track, against which all musicians can perform.
These and other features and advantages of the invention will be more readily apparent upon reading the following description of a preferred exemplified embodiment of the invention and upon reference to the accompanying drawings wherein: